Bayesian Reinforcement Learning Research Articles

Bayesian Reinforcement Learning With Limited Cognitive Load.

All biological and artificial agents must act given limits on their ability to acquire and process information. As such, a general theory of adaptive behavior should be able to account for the complex interactions between an agent's learning history, decisions, and capacity constraints. Recent work in computer science has begun to clarify the principles that shape these dynamics by bridging ideas from reinforcement learning, Bayesian decision-making, and rate-distortion theory. This body of work provides an account of capacity-limited Bayesian reinforcement learning, a unifying normative framework for modeling the effect of processing constraints on learning and action selection. Here, we provide an accessible review of recent algorithms and theoretical results in this setting, paying special attention to how these ideas can be applied to studying questions in the cognitive and behavioral sciences.

Optimization of laser annealing parameters based on bayesian reinforcement learning

AbstractDeveloping new semiconductor processes consumes tremendous time and cost. Therefore, we applied Bayesian reinforcement learning (BRL) with the assistance of technology computer-aided design (TCAD). The fixed or variable prior BRL is tested where the TCAD prior is fixed or is changed by the experimental sampling and decays during the entire RL procedure. The sheet resistance (Rs) of the samples treated by laser annealing is the optimization target. In both cases, the experimentally sampled data points are added to the training dataset to enhance the RL agent. The model-based experimental agent and a model-free TCAD Q-Table are used in this study. The results of BRL proved that it can achieve lower Rs minimum values and variances at different hyperparameter settings. Besides, two action types, i.e., point to state and increment of levels, are proven to have similar results, which implies the method used in this study is insensitive to the different action types.

Bayesian reinforcement learning: A basic overview

We and other animals learn because there is some aspect of the world about which we are uncertain. This uncertainty arises from initial ignorance, and from changes in the world that we do not perfectly know; the uncertainty often becomes evident when our predictions about the world are found to be erroneous. The Rescorla-Wagner learning rule, which specifies one way that prediction errors can occasion learning, has been hugely influential as a characterization of Pavlovian conditioning and, through its equivalence to the delta rule in engineering, in a much wider class of learning problems. Here, we review the embedding of the Rescorla-Wagner rule in a Bayesian context that is precise about the link between uncertainty and learning, and thereby discuss extensions to such suggestions as the Kalman filter, structure learning, and beyond, that collectively encompass a wider range of uncertainties and accommodate a wider assortment of phenomena in conditioning.

Bayesian reinforcement learning reliability analysis

A Bayesian reinforcement learning reliability method that combines Bayesian inference for the failure probability estimation and reinforcement learning-guided sequential experimental design is proposed. The reliability-oriented sequential experimental design is framed as a finite-horizon Markov decision process (MDP), with the associated utility function defined by a measure of epistemic uncertainty about Kriging-estimated failure probability, referred to as integrated probability of misclassification (IPM). On this basis, a one-step Bayes optimal learning function termed integrated probability of misclassification reduction (IPMR), along with a compatible convergence criterion, is defined. Three effective strategies are implemented to accelerate IPMR-informed sequential experimental design: (i) Analytical derivation of the inner expectation in IPMR, simplifying it to a single expectation. (ii) Substitution of IPMR with its upper bound IPMRU to avoid element-wise computation of its integrand. (iii) Rational pruning of both quadrature set and candidate pool in IPMRU to alleviate computer memory constraint. The efficacy of the proposed approach is demonstrated on two benchmark examples and two numerical examples. Results indicate that IPMRU facilitates a much more rapid reduction of IPM compared to other existing learning functions, while requiring much less computational time than IPMR itself. Therefore, the proposed reliability method offers a substantial advantage in both computational efficiency and accuracy, especially in complex dynamic reliability problems.

Decentralized fused-learner architectures for Bayesian reinforcement learning

Decentralized training is a robust solution for learning over an extensive network of distributed agents. Many existing solutions involve the averaging of locally inferred parameters which constrain the architecture to independent agents with identical learning algorithms. Here, we propose decentralized fused-learner architectures for Bayesian reinforcement learning, named fused Bayesian-learner architectures (FBLAs), that are capable of learning an optimal policy by fusing potentially heterogeneous Bayesian policy gradient learners, i.e., agents that employ different learning architectures to estimate the gradient of a control policy. The novelty of FBLAs relies on fusing the full posterior distributions of the local policy gradients. The inclusion of higher-order information, i.e., probabilistic uncertainty, is employed to robustly fuse the locally-trained parameters. FBLAs find the barycenter of all local posterior densities by minimizing the total Kullback–Leibler divergence from the barycenter distribution to the local posterior densities. The proposed FBLAs are demonstrated on a sensor-selection problem for Bernoulli tracking, where multiple sensors observe a dynamic target and only a subset of sensors is allowed to be active at any time.

A Bayesian Deep Reinforcement Learning-Based Resilient Control for Multi-Energy Micro-Gird

Aiming at a cleaner future power system, many regimes in the world have proposed their ambitious decarboniz-ing plan, with increasing penetration of renewable energy sources (RES) playing an alternative role to conventional energy. As a re-sult, power system tends to have less backup capacity and operate near their designed limit, thus exacerbating system vulnerability against extreme events. Under this reality, resilient control for the multi-energy micro-grid is facing the following challenges, which are: 1) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">the effect from the stochastic uncertainties of</i> RES; 2) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">the need for a model-free and fast-reacting control scheme under extreme events; and</i> 3) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">efficient exploration and robust performance with limited extreme events data</i> . <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">To deal with the aforementioned challenges, this paper pro-poses a novel Bayesian Deep Reinforcement Learning-based resilient control approach for multi-energy micro-grid. In partic-ular, the proposed approach replaces the deterministic network in traditional Reinforcement Learning with a Bayesian probabilistic network in order to obtain an approximation of the value function distribution, which effectively solves the Q-value overestimation issue. Compared with the naive Deep Deterministic Policy Gra-dient (DDPG) method and optimization method, the effectiveness and importance of employing the Bayesian Reinforcement Learn-ing approach are investigated and illustrated across different operating scenarios. Case studies have shown that by using the Monte Carlo posterior mean of the Bayesian value function distribution instead of a deterministic estimation, the proposed Bayesian Deep Deterministic Policy Gradient (BDDPG) method achieves a near-optimum policy in a more stable process, which verifies the robustness and the practicability of the proposed approach.

A Bayesian reinforcement learning approach in markov games for computing near-optimal policies

A Bayesian reinforcement learning approach in markov games for computing near-optimal policies

Toward Personalization of User Preferences in Partially Observable Smart Home Environments

The technologies used in smart homes have recently improved to learn the user preferences from feedback in order to enhance the user convenience and quality of experience. Most smart homes learn a uniform model to represent the thermal preferences of users, which generally fails when the pool of occupants includes people with different sensitivities to temperature, for instance, due to age and physiological factors. Thus, a smart home with a single optimal policy may fail to provide comfort when a new user with a different preference is integrated into the home. In this paper, we propose a Bayesian Reinforcement learning framework that can approximate the current occupant state in a partially observable smart home environment using its thermal preference, and then identify the occupant as a new user or someone is already known to the system. Our proposed framework can be used to identify users based on the temperature and humidity preferences of the occupant when performing different activities to enable personalization and improve comfort. We then compare the proposed framework with a baseline long short-term memory learner that learns the thermal preference of the user from the sequence of actions that it takes. We perform these experiments with up to 5 simulated human models each based on hierarchical reinforcement learning. The results show that our framework can approximate the belief state of the current user just by its temperature and humidity preferences across different activities with a high degree of accuracy.

Model-based Bayesian reinforcement learning for enhancing primary user performance under jamming attack

Due to the broadcast nature of wireless networks, jamming attacks are inevitable. Therefore, this paper considers enhancing the throughput of a primary user (PU) suffering from a random jamming attack near the receiver in a cognitive radio network. A cooperative spectrum-sharing scheme is proposed between the PU and an energy-constrained secondary user (SU) harvesting non-RF energy. The SU location is optimized to maximize the PU throughput. Then, a decision problem is formulated to adaptively optimize the PU decision regarding the cooperation with SU and maximize the PU throughput in the long run, considering the uncertainties about the jammer activity and the SU’s available energy. Unlike the previous work, our scheme assumes no prior knowledge about the parameters controlling the jammer’s random behavior, the energy harvesting process, and the sensing system. Bayesian reinforcement learning (BRL) is adopted to learn the unknown parameters and optimize the PU decisions simultaneously. The Bayes-adaptive mixed-observable Markov decision process (BAMOMDP) model is proposed to extend the current BRL models to handle the state’s mixed observability. Simulation results show the efficiency and robustness of the proposed scheme in enhancing the PU/SU throughputs and its ability to adapt to non-stationary environments. Compared with previous work, the proposed scheme can improve the PU average throughput by 15% in non-stationary environments and up to 40% in stationary environments with low jamming sensing capabilities.

Energy optimization of wind turbines via a neural control policy based on reinforcement learning Markov chain Monte Carlo algorithm

This study focuses on the numerical analysis and optimal control of vertical-axis wind turbines (VAWT) using Bayesian reinforcement learning (RL). We specifically address small-scale wind turbines, which are well-suited to local and compact production of electrical energy on a small scale, such as urban and rural infrastructure installations. Existing literature concentrates on large scale wind turbines which run in unobstructed, mostly constant wind profiles. However urban installations generally must cope with rapidly changing wind patterns. To bridge this gap, we formulate and implement an RL strategy using the Markov chain Monte Carlo (MCMC) algorithm to optimize the long-term energy output of a wind turbine. Our MCMC-based RL algorithm is a model-free and gradient-free algorithm, in which the designer does not have to know the precise dynamics of the plant and its uncertainties. Our method addresses the uncertainties by using a multiplicative reward structure, in contrast with additive reward used in conventional RL approaches. We have shown numerically that the method specifically overcomes the shortcomings typically associated with conventional solutions, including, but not limited to, component aging, modeling errors, and inaccuracies in the estimation of wind speed patterns. Our results show that the proposed method is especially successful in capturing power from wind transients; by modulating the generator load and hence the rotor torque load, so that the rotor tip speed quickly reaches the optimum value for the anticipated wind speed. This ratio of rotor tip speed to wind speed is known to be critical in wind power applications. The wind to load energy efficiency of the proposed method was shown to be superior to two other methods; the classical maximum power point tracking method and a generator controlled by deep deterministic policy gradient (DDPG) method.

Reinforcement Learning Under Uncertainty: Expected Versus Unexpected Uncertainty and State Versus Reward Uncertainty

Two prominent types of uncertainty that have been studied extensively are expected and unexpected uncertainty. Studies suggest that humans are capable of learning from reward under both expected and unexpected uncertainty when the source of variability is the reward. How do people learn when the source of uncertainty is the environment’s state and the rewards themselves are deterministic? How does their learning compare with the case of reward uncertainty? The present study addressed these questions using behavioural experimentation and computational modelling. Experiment 1 showed that human subjects were generally able to use reward feedback to successfully learn the task rules under state uncertainty, and were able to detect a non-signalled reversal of stimulus-response contingencies. Experiment 2, which combined all four types of uncertainties—expected versus unexpected uncertainty, and state versus reward uncertainty—highlighted key similarities and differences in learning between state and reward uncertainties. We found that subjects performed significantly better in the state uncertainty condition, primarily because they explored less and improved their state disambiguation. We also show that a simple reinforcement learning mechanism that ignores state uncertainty and updates the state-action value of only the identified state accounted for the behavioural data better than both a Bayesian reinforcement learning model that keeps track of belief states and a model that acts based on sampling from past experiences. Our findings suggest a common mechanism supports reward-based learning under state and reward uncertainty.

Trace-class Gaussian priors for Bayesian learning of neural networks with MCMC

AbstractThis paper introduces a new neural network based prior for real valued functions. Each weight and bias of the neural network has an independent Gaussian prior, with the key novelty that the variances decrease in the width of the network in such a way that the resulting function is well defined in the limit of an infinite width network. We show that the induced posterior over functions is amenable to Monte Carlo sampling using Hilbert space Markov chain Monte Carlo (MCMC) methods. This type of MCMC is stable under mesh refinement, i.e. the acceptance probability does not degenerate as more parameters of the function's prior are introduced, even ad infinitum. We demonstrate these advantages over other function space priors, for example in Bayesian Reinforcement Learning.

Social learning across adolescence: A Bayesian neurocognitive perspective

Adolescence is a period of social re-orientation in which we are generally more prone to peer influence and the updating of our beliefs based on social information, also called social learning, than in any other stage of our life. However, how do we know when to use social information and whose information to use and how does this ability develop across adolescence? Here, we review the social learning literature from a behavioral, neural and computational viewpoint, focusing on the development of brain systems related to executive functioning, value-based decision-making and social cognition. We put forward a Bayesian reinforcement learning framework that incorporates social learning about value associated with particular behavior and uncertainty in our environment and experiences. We discuss how this framework can inform us about developmental changes in social learning, including how the assessment of uncertainty and the ability to adaptively discriminate between information from different social sources change across adolescence. By combining reward-based decision-making in the domains of both informational and normative influence, this framework explains both negative and positive social peer influence in adolescence.

Regularization Guarantees Generalization in Bayesian Reinforcement Learning through Algorithmic Stability

In the Bayesian reinforcement learning (RL) setting, a prior distribution over the unknown problem parameters -- the rewards and transitions -- is assumed, and a policy that optimizes the (posterior) expected return is sought. A common approximation, which has been recently popularized as meta-RL, is to train the agent on a sample of N problem instances from the prior, with the hope that for large enough N, good generalization behavior to an unseen test instance will be obtained. In this work, we study generalization in Bayesian RL under the probably approximately correct (PAC) framework, using the method of algorithmic stability. Our main contribution is showing that by adding regularization, the optimal policy becomes uniformly stable in an appropriate sense. Most stability results in the literature build on strong convexity of the regularized loss -- an approach that is not suitable for RL as Markov decision processes (MDPs) are not convex. Instead, building on recent results of fast convergence rates for mirror descent in regularized MDPs, we show that regularized MDPs satisfy a certain quadratic growth criterion, which is sufficient to establish stability. This result, which may be of independent interest, allows us to study the effect of regularization on generalization in the Bayesian RL setting.

Content-Based Music Recommendation Using Non-Stationary Bayesian Reinforcement Learning

This paper presents a music recommendation system for the offline libraries of songs that employs the concepts of reinforcement learning to obtain satisfactory recommendations based on the various considered content-based parameters. In order to obtain insights about the effectiveness of the generated recommendations, parallel instances of single-play multi-arm bandit algorithms are maintained. In conjunction to this, the concepts of Bayesian learning are considered to model the user preferences, by assuming the environment’s reward generating process to be non-stationary and stochastic. The system is designed to be simple, easy to implement, and at-par with the user satisfaction, within the bounds of the input data capabilities.

Content-Based Music Recommendation Using Non-Stationary Bayesian Reinforcement Learning

This paper presents a music recommendation system for the offline libraries of songs that employs the concepts of reinforcement learning to obtain satisfactory recommendations based on the various considered content-based parameters. In order to obtain insights about the effectiveness of the generated recommendations, parallel instances of single-play multi-arm bandit algorithms are maintained. In conjunction to this, the concepts of Bayesian learning are considered to model the user preferences, by assuming the environment’s reward generating process to be non-stationary and stochastic. The system is designed to be simple, easy to implement, and at-par with the user satisfaction, within the bounds of the input data capabilities.

Content-Based Music Recommendation Using Non-Stationary Bayesian Reinforcement Learning

This paper presents a music recommendation system for the offline libraries of songs that employs the concepts of reinforcement learning to obtain satisfactory recommendations based on the various considered content-based parameters. In order to obtain insights about the effectiveness of the generated recommendations, parallel instances of single-play multi-arm bandit algorithms are maintained. In conjunction to this, the concepts of Bayesian learning are considered to model the user preferences, by assuming the environment’s reward generating process to be non-stationary and stochastic. The system is designed to be simple, easy to implement, and at-par with the user satisfaction, within the bounds of the input data capabilities.

Fog Computing Resource Management for Video Processing Using Evolutionary Bayesian Optimization and Nondeterministic Deep Reinforcement Learning

Fog Computing Resource Management for Video Processing Using Evolutionary Bayesian Optimization and Nondeterministic Deep Reinforcement Learning

Association of Environmental Uncertainty With Altered Decision-making and Learning Mechanisms in Youths With Obsessive-Compulsive Disorder

Adults with obsessive-compulsive disorder (OCD) display perseverative behavior in stable environments but exhibit vacillating choice when payoffs are uncertain. These findings may be associated with intolerance of uncertainty and compulsive behaviors; however, little is known about the mechanisms underlying learning and decision-making in youths with OCD because research into this population has been limited. To investigate cognitive mechanisms associated with decision-making in youths with OCD by using executive functioning tasks and computational modeling. In this cross-sectional study, 50 youths with OCD (patients) and 53 healthy participants (controls) completed a probabilistic reversal learning (PRL) task between January 2014 and March 2020. A separate sample of 27 patients and 46 controls completed the Wisconsin Card Sorting Task (WCST) between January 2018 and November 2020. The study took place at the University of Cambridge in the UK. Decision-making mechanisms were studied by fitting hierarchical bayesian reinforcement learning models to the 2 data sets and comparing model parameters between participant groups. Model parameters included reward and punishment learning rates (feedback sensitivity), reinforcement sensitivity and decision consistency (exploitation), and stickiness (perseveration). Associations of receipt of serotonergic medication with performance were assessed. In total, 50 patients (29 female patients [58%]; median age, 16.6 years [IQR, 15.3-18.0 years]) and 53 controls (30 female participants [57%]; median age, 16.4 years [IQR, 14.8-18.0 years]) completed the PRL task. A total of 27 patients (18 female patients [67%]; median age, 16.1 years [IQR, 15.2-17.2 years]) and 46 controls (28 female participants [61%]; median age, 17.2 [IQR, 16.3-17.6 years]) completed the WCST. During the reversal phase of the PRL task, patients made fewer correct responses (mean [SD] proportion: 0.83 [0.16] for controls and 0.61 [0.31] for patients; 95% CI, -1.31 to -0.64) and switched choices more often following false-negative feedback (mean [SD] proportion: 0.09 [0.16] for controls vs 0.27 [0.34] for patients; 95% CI, 0.60-1.26) and true-positive feedback (mean [SD] proportion: 0.93 [0.17] for controls vs 0.73 [0.34] for patients; 95% CI, -2.17 to -1.31). Computational modeling revealed that patients displayed enhanced reward learning rates (mean difference [MD], 0.21; 95% highest density interval [HDI], 0.04-0.38) but decreased punishment learning rates (MD, -0.29; 95% HDI, -0.39 to -0.18), reinforcement sensitivity (MD, -4.91; 95% HDI, -9.38 to -1.12), and stickiness (MD, -0.35; 95% HDI, -0.57 to -0.11) compared with controls. There were no group differences on standard WCST measures and computational model parameters. However, patients who received serotonergic medication showed slower response times (mean [SD], 1420.49 [279.71] milliseconds for controls, 1471.42 [212.81] milliseconds for patients who were unmedicated, and 1738.25 [349.23] milliseconds for patients who were medicated) (control vs medicated MD, -320.26 [95% CI, -547.00 to -88.68]) and increased unique errors (mean [SD] proportion: 0.001 [0.004] for controls, 0.002 [0.004] for patients who were unmedicated, and 0.008 [0.01] for patients who were medicated) (control vs medicated MD, -0.007 [95% CI, -3.14 to -0.36]) on the WCST. The results of this cross-sectional study indicated that youths with OCD showed atypical probabilistic reversal learning but were generally unimpaired on the deterministic WCST, although unexpected results were observed for patients receiving serotonergic medication. These findings have implications for reframing the understanding of early-onset OCD as a disorder in which decision-making is associated with uncertainty in the environment, a potential target for therapeutic treatment. These results provide continuity with findings in adults with OCD.

Cost-Optimized Microgrid Coalitions Using Bayesian Reinforcement Learning

Microgrids are empowered by the advances in renewable energy generation, which enable the microgrids to generate the required energy for supplying their loads and trade the surplus energy to other microgrids or the macrogrid. Microgrids need to optimize the scheduling of their demands and energy levels while trading their surplus with others to minimize the overall cost. This can be affected by various factors such as variations in demand, energy generation, and competition among microgrids due to their dynamic nature. Thus, reaching optimal scheduling is challenging due to the uncertainty caused by the generation/consumption of renewable energy and the complexity of interconnected microgrids and their interplay. Previous works mainly rely on modeling-based approaches and the availability of precise information on microgrid dynamics. This paper addresses the energy trading problem among microgrids by minimizing the cost while uncertainty exists in microgrid generation and demand. To this end, a Bayesian coalitional reinforcement learning-based model is introduced to minimize the energy trading cost among microgrids by forming stable coalitions. The results show that the proposed model can minimize the cost up to 23% with respect to the coalitional game theory model.